STATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS. x, where. = y - ˆ " 1

Size: px

Start display at page:

Download "STATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS. x, where. = y - ˆ " 1"

Arline Miles
5 years ago
Views:

1 STATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS Recall Assumpto E(Y x) η 0 + η x (lear codtoal mea fucto) Data (x, y ), (x 2, y 2 ),, (x, y ) Least squares estmator ˆ E (Y x) ˆ " 0 + ˆ " x, where ˆ " SXY ˆ " 0 y - ˆ " x ( x - x ) 2 x ( x - x ) SXY ( x - x ) (y - y ) ( x - x ) y Commets So far we have estmates of the parameters η 0 ad η, but have o dea how good these estmates are 2 If our data were the etre populato, we could also use the same least squares procedure to ft a approxmate le to the codtoal sample meas 3 If we have a smple radom sample from the populato ad also assume that e x (equvaletly, Y x) s ormal wth costat varace, the the least squares estmates are the same as the maxmum lkelhood estmates of η 0 ad η Propertes of ˆ " 0 ad ˆ " ) ˆ " SXY (x " x )y where c (x " x ) (x " x ) y " c y Thus If the x 's are fxed (as the blood lactc acd example), the ˆ " s a lear combato of the y 's Note Here we wat to thk of each y as a observato from a radom varable Y wth dstrbuto Y x Sayg that the y s are depedet s the same as sayg that the Y s are depedet radom varables Thus, f the y s are depedet ad each Y x s ormal, the ˆ " s also ormal If the Y x 's are ot ormal but s large, the ˆ " s approxmately ormal Ths wll allow us to do ferece o ˆ " (Detals later) 2) c (x " x ) (x " x ) 0 (as see establshg the alterate expresso for )

2 2 3) x c x (x " x ) x (x " x ) Remark Recall the somewhat aalogous propertes for the resduals ˆ e 4) ˆ " 0 y - ˆ " x hece " y - c y " x ( " c x )y, also a lear combato of the y 's, 5) The sum of the coeffcets (4) s ( " c x ) ( ) " x c ( ) " x 0 Samplg dstrbutos of ˆ " 0 ad ˆ " Cosder x,, x as fxed (e, codto o x,, x ) Model Assumptos ("The" Smple Lear Regresso Model Verso 2) E(Y x) η 0 + η x (lear codtoal mea fucto) 2 (NEW) Var(Y x) σ 2 (Equvaletly, Var(e x) σ 2 ) (costat varace) 3 (NEW) y,, y are depedet observatos (depedece) The ew assumpto meas we ca cosder y,, y as comg from depedet radom varables Y,, Y, where Y has the dstrbuto of Y x Commet We do ot assume that the x 's are dstct If, for example, x x 2, the we are assumg that y ad y 2 are depedet observatos from the same codtoal dstrbuto Y x Sce Y,, Y are radom varables, so s ˆ " -- but t depeds o the choce of x,, x, so we ca talk about the codtoal dstrbuto ˆ " x,, x Expected value of ˆ " (as the y 's vary) E( ˆ " x,, x ) E( " c Y x,, x ) Thus ˆ " s a ubased estmator of η c E(Y x,, x ) c E(Y x ) (sce Y depeds oly o x ) c (η 0 + η x ) (model assumpto) η 0 c + η c x η η η

3 3 Varace of ˆ " (as the y 's vary) Var( ˆ " x,, x ) Var( " c Y x,, x ) c 2 Var(Y x,, x ) c 2 Var(Y x ) (sce Y depeds oly o x ) c 2 σ 2 σ 2 2 c σ 2 (x " x ) 2 & % ( (defto of c $ ' ) " 2 $ (x ( ) 2 x ) 2 " 2 For short Var( ˆ " ) " 2 SD( ˆ " ) " Commets Ths s vaguely aalogous to the samplg stadard devato for a mea y populato stadard devato SD(estmator) somethg However, here the "somethg," amely, s more complcated But we ca stll aalyze ths formula to see how the stadard devato vares wth the codtos of samplg For y, the deomator s the square root of, so we see that as becomes larger, the samplg stadard devato of y gets smaller Here, recallg that ( x - x ) 2, we reaso that If the x 's are far from x (e, spread out), s, so SD( ˆ " ) s If the x 's are close to x (e, close together), s, so SD( ˆ " ) s Thus f you are desgg a expermet, choosg the x 's to be from ther mea wll result a more precse estmate of ˆ " (Assumg all the model codtos ft!) Expected value ad varace of ˆ " 0 Usg the formula ˆ " 0 ( " c x )y, calculatos (left to the terested studet) smlar to those for ˆ " wll show E( ˆ " 0 ) η 0 (So ˆ " 0 s a ubased estmator of η 0 )

4 4 Var ( ˆ " 0 ) " 2 + x 2 & % $ ( ', so SD( ˆ " 0 ) " + x 2 Aalyzg the varace formula A larger x gves a varace for ˆ " 0 Does ths agree wth tuto? A larger sample sze teds to gve a varace for ˆ " 0 The varace of ˆ " 0 s (except whe x < ) tha the varace of ˆ " Does ths agree wth tuto? The spread of the x 's affects the varace of ˆ " 0 the same way t affects the varace of ˆ " Covarace of ˆ " 0 ad ˆ " Smlar calculatos (left to the terested studet) wll show x Cov( ˆ " 0, ˆ " ) " 2 Thus ˆ " 0 ad ˆ " are ot depedet (except possbly whe ) Does ths agree wth tuto? The sg of Cov( ˆ " 0, ˆ " ) s opposte that of x Does ths agree wth tuto? Estmatg σ 2 To use the varace formulas above for ferece, we eed to estmate σ 2 ( Var(Y x ), the same for all ) Frst, some plausble reasog If we had lots of observatos y, y 2,,y m from Y x, the we could use the uvarate stadard devato m (y m " j " y ) 2 j of these m observatos to estmate σ 2 (Here y s the mea of y, y 2,,y m, whch would be our best estmate of E(Y x ) just usg y, y 2,,y m ) We do't typcally have lots of y's from oe x, so we mght try (reasog that E ˆ (Y x ) ) s our best estmate of E(Y x )) [y " " E ˆ (Y x )] 2

5 5 " 2 e ˆ " RSS However (just as the uvarate case, we eed a deomator - to get a ubased estmator), a legthy calculato (omtted) wll show that E(RSS x,, x ) (-2) σ 2 (where the expected value s over all samples of the y 's wth the x 's fxed) Thus we use the estmate ˆ " 2 " 2 RSS to get a ubased estmator for σ 2 E( ˆ " 2 x,, x ) σ 2 [If you lke to thk heurstcally terms of losg oe degree of freedom for each calculato from data volved the estmator, ths makes sese Both ˆ " 0 ad ˆ " eed to be calculated from the data to get RSS] Stadard Errors for ˆ " 0 ad ˆ " Usg ˆ " RSS " 2 as a estmate of σ the formulas for SD( ˆ " 0 ) ad SD( ˆ " ), we obta the stadard errors ad se ( ˆ " ) se( ˆ " 0 ) ˆ " ˆ " + x 2 as estmates of SD( ˆ " ) ad SD( ˆ " 0 ), respectvely

Lecture 7. Confidence Intervals and Hypothesis Tests in the Simple CLR Model

Lecture 7. Confidence Intervals and Hypothesis Tests in the Simple CLR Model Lecture 7. Cofdece Itervals ad Hypothess Tests the Smple CLR Model I lecture 6 we troduced the Classcal Lear Regresso (CLR) model that s the radom expermet of whch the data Y,,, K, are the outcomes. The